On the Approximation Rate of HierarchicalMixtures - of - Experts for Generalized

We investigate a class of hierarchical mixtures-of-experts (HME) models where generalized linear models with nonlinear mean functions of the form (+ x T) are mixed. Here () is the inverse link function. It is shown that mixtures of such mean functions can approximate a class of smooth functions of the form (h(x)) where h() 2 W 1 2;K (a Sobolev class over 0; 1] s), as the number of experts m in the network increases. An upper-bound of the approximation rate is given as O(m ?2=s) in L p norm. This rate can be achieved within the family of HME structures with no more than s-layers, where s is the dimension of the predictor x.